DRAFT: Scalability Bugs: When 100-Node Testing is Not Enough

We highlight the problem of scalability bugs, a new class of bugs that appear in “cloud-scale” distributed systems. Scalability bugs are latent bugs that are clusterscale dependent, whose symptoms typically surface in large-scale deployments, but not in small or mediumscale deployments. The standard practice to test large distributed systems is to deploy them on a large number of machines (“real-scale testing”), which is difficult and expensive. New methods are needed to reduce developers’ burdens in finding, reproducing, and debugging scalability bugs. We propose “scale check,” an approach that helps developers find and replay scalability bugs at real scales, but do so only on one machine and still achieve a high accuracy (i.e., similar observed behaviors as if the nodes are deployed in real-scale testing).